AFOSR  Research  Programs  in  Image  Fusion 

J.  Tangney 

Directorate  of  Chemistry  and  Life  Sciences 
Air  Force  Office  of  Scientific  Research 
Arlington,  VA,  U.S.A. 

John.Tangney@afosr.af.mil 


Abstract  -  The  U.S.  Air  Force  Office  of  Scientific 
Research  supports  basic  research  in  image  fusion  within 
several  programs.  These  programs  are  presented  in 
context  of  Air  Force  technology  needs  for  targeting, 
image  exploitation,  and  autonomous  systems.  Programs 
include  research  involving  human  perception  and  neural 
processing  in  other  biological  systems,  algorithms  for 
fusion  from  multiple  sources  and  platforms,  and  novel 
sensors.  Available  mechanisms  for  support  of 
collaborative  research  will  also  be  presented. 
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1  Introduction 

The  Air  Force  Office  of  Scientific  Research  (AFOSR)  is 
the  single  manager  of  basic  research  for  the  U.S.  Air 
Force  (USAF).  Basic  research  is  defined  in  the  U.S. 
Defense  Department  as  the  most  fundamental  of  work 
along  a  research  and  development  spectrum  from 
knowledge  discovery  to  technology  demonstration.  The 
AFOSR  supports  many  areas  of  science,  but  focuses  its 
support  on  topics  expected  strongly  to  contribute  to  the 
technology  needs  of  the  USAF.  These  programs  are 
executed  largely  through  grants  and  contracts  to 
industrial  and  academic  researchers.  Description  of 
current  topics  of  AFOSR  support  is  available  at  the 
AFOSR  home  page  (http://www.afosr.af.mil). 

Fusion,  the  topic  of  this  meeting,  is  not  an  AFOSR 
focus  area  because  AFOSR  defines  topics  in  scientific 
terms,  not  technological  ones.  Nevertheless,  several 
AFOSR  programs  contribute  to  fusion  technologies.  I 
will  describe  program  contributions  each  in  turn, 
emphasizing  those  that  take  advantage  of  biological 
signal  processing,  of  special  interest  to  me. 

1.1  AFRL  Fusion  Science  and  Technology 

AFOSR  is  a  part  of  the  Air  Force  Research  Laboratory 
(AFRL)  and  uses  its  scientific  programs  to  support 
technology  developments  within  AFRL's  several 
technology  directorates.  These  technology  directorates 
conduct  some  basic  research,  but  have  primary 
responsibilities  for  technology  development  and 
application.  Some  of  the  fusion-related  work  in  the 


Sensors  Directorate  (SN)  and  the  Information  Directorate 
(IF)  will  be  illustrated.  For  example,  these  two 
directorates  of  AFRL  work  together  on  a  “fusion 
network”  for  technology  demonstrations  that  integrate 
sensors  and  processing.  AFOSR  contributes  to  SN 
technology  advances  in  novel  sensors,  modeling  of  the 
environment,  data  storage  and  processing.  AFOSR  also 
contributes  to  IF  technology  advances  in  data  and 
information  fusion  and  image  exploitation. 

Through  the  AFRL  technology  directorates,  AFOSR 
funded  science  is  grounded  in  applications  of  interest  to 
the  Air  Force.  In  addition  to  SN  and  IF,  other 
directorates  of  the  AFRL  contribute  to  fusion 
technologies.  Some  information  on  the  work  of  each  of 
the  technology  directorates  is  available  at 
http://www.afrl.af.mil 

Taken  as  a  whole,  however,  the  AFRL  technology 
effort  related  to  fusion  includes  all  the  elements  required: 
anthropomorphizing  somewhat,  effective  eyes  and  hands 
connected  with  a  brain.  The  sensing  goal  is  development 
of  a  common  operating  picture  robust  with  respect  to 
sensor  technologies,  and  the  effecting  goal  is  for  speed 
and  accuracy  of  threat  response. 

1.2  AFRL  Partnering  in  Fusion  Technologies 

The  AFRL  also  works  closely  with  other  military 
departments,  with  industry  and  other  government 
agencies.  The  AFRL  maintains  a  virtual  distributed 
laboratory,  of  which  the  earlier  mentioned  fusion  network 
is  a  part,  to  coordinate  technology  developments  across 
the  institutions  involved. 

The  full  range  of  fusion  technologies  is  of  interest  to 
the  Air  Force,  and  the  AFRL  is  working  in  a  leadership 
role  on  many,  if  not  most,  of  them.  This  role  requires 
coordinating  efforts  of  many  research  and  development 
organizations.  The  science  supported  by  AFOSR  is 
conducted  in  many  of  these  organizations  and,  when  not, 
is  connected  tightly  to  them  through  the  AFRL. 

2  Fusion  Science 

The  AFOSR  supports  research  in  areas  related  to  fusion 
of  information  and  data,  including  multi-spectral  image 
sensing,  fusion  and  processing  for  recognition  and 
identification  of  targets,  as  well  as  integrated  distributed 
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parameter  control  of  the  vehicles  involved  in  these  tasks. 
Main  topics  of  research  are  considered  in  turn. 

2.1  Sensors 

The  AFOSR  supports  research  on  sensing  materials  and 
devices  for  multi-spectral  imaging.  Some  research 
supported  in  Biological  programs  takes  inspiration  from 
study  of  natural  systems.  The  python,  for  example,  can 
fuse  the  visible  and  infrared  spectra  to  detect  and  capture 
its  prey.  Research  supported  in  this  program  has  recently 
determined  [1]  that  the  heat  sensitive  organ  of  the  python 
may  take  advantage  of  specialized  membrane  structures 
that  provide  a  degree  of  spectral  filtering  for  underlying 
heat  transduction  mechanisms.  The  microstructure  of  the 
overlaid  membrane,  specifically,  the  spatial  distribution 
of  submicron  pits  within  the  membrane,  is  suspected  to 
play  a  role  in  spectral  filtering  and  is  the  object  of  current 
study.  The  overall  goal  of  this  research  is  to  determine 
whether  infrared  or  heat  sensitive  detectors  might  be 
operated  at  ambient  temperatures  with  higher  sensitivity 
and  resolution  than  those  available  using  current 
technologies. 


Figure  1  :  Rattlers  and  Pythons  Sense  Visible  and 
Infrared 


In  the  optoelectronic  materials  program,  AFOSR 
supports  research  on  sensor  materials  for  focal  plane 
arrays  and  other  devices,  to  cover  the  entire  spectrum 
with  fast,  sensitive,  high  resolution  images.  Current 
approaches  include  quantum  well  infrared  photodetection 
and  quantum  dot  arrays.  These  approaches  provide  fast 
(picoseconds),  sensitive  (tens  of  milli-Kelvin),  high 
density  sensor  arrays  for  a  variety  of  applications. 

The  AFOSR  optoelectronics  materials  program  also 
supports  research  on  integrated  focal  plan  imaging  of 
multiple  spectral  bands.  The  interspersed  spectral 
images  that  result  offer  advantages  in  image  registration 
and  sensor  fusion  because  images  in  multiple  bands  are 
obtained  at  the  same  time  through  the  same  optics. 

One  approach  to  integrated  multi-spectral  sensing 
involves  construction  of  three  by  three  pixel  subarrays  in 
which  each  of  the  nine  sensing  sub-pixels  receives  an 
image  filtered  into  one  of  three  infrared  bands  and  one  of 
three  polarimetric  orientations.  Polarimetric  images  are 
of  interest  because  man-made  objects,  with  their  planar 
surfaces,  typically  do  not  reflect  light  of  all  polarizations. 
Fused  images  can  also  be  constructed  to  provide  system 
operators  and  image  analysts  with  new  information  that 


may  be  useful  in  separating  man-made  objects  from 
natural  backgrounds  —  camouflage  breaking. 

In  the  approach  described  above,  the  infrared  sensing 
material  is  obtained  from  novel  3-5  materials,  spectral 
tuning  is  obtained  using  interference  filters  constructed 
of  Germanium  and  Silicon  Oxide,  and  polarimetric 
sensitivity  is  obtained  using  a  standard  grating  technique. 

AFOSR  also  supports  research  to  combine  multi-band 
optical  and  radar  sensing  elements  into  compact,  low 
power,  low  weight  modules.  One  current  approach 
involves  research  at  the  fundamental  device  design  level 
to  develop  common  aperture  devices  for  active  and 
passive  sensing  of  EO  and  RF.  According  to  one  notion 
for  active  modes,  for  example,  LADAR  illumination 
would  be  modulated  at  mm-wave  frequencies  so  that 
return  light  might  be  detected  simultaneously  and 
demodulated  with  the  radar  returns  for  precise  pixel 
registration  of  the  two  images. 

It  has  been  long  known  that  interaction  can  occur 
between  electromagnetic  fields  and  various  material 
excitations.  Brillouin  scattering  results  from  the 
coupling  of  electromagnetic  fields  and  acoustic  waves  in 
matter.  For  the  first  time,  as  far  as  we  know,  researchers 
working  with  the  Air  Force  Office  of  Scientific  Research 
are  exploiting  this  electromagnetic-acoustic  coupling  to 
attempt  to  extend  Air  Force  imaging  capabilities. 

In  a  real  material  medium  such  as  foliage,  wood, 
soil,  concrete,  or  plastic,  the  material  polarizability  and 
material  conductivity  are  dependent  on  the  local  pressure. 
Material  polarizability  refers  to  the  degree  to  which  fixed 
charge  in  the  material  can  be  distorted  by  an  electric 
field.  A  measure  of  material  polarizability  is  the 
dielectric  constant.  Material  conductivity  refers  to  the 
tendency  of  free  charges  within  the  material  to  move  in 
the  presence  of  an  electric  field.  While  polarizability  and 
conductivity  are  functions  of  local  pressure,  these 
material  properties  also  influence  electromagnetic  wave 
speed  in  the  medium.  Thus,  there  is  a  connection 
between  pressure  and  the  movement  of  electromagnetic 
energy,  and,  it  is  this  interaction  that  is  being  exploited 
to  enhance  imaging. 

Professor  Thomas  Banks  [2]  and  colleagues  at  the 
Center  for  Scientific  Computation  at  NC  State 
University,  under  AFOSR  sponsorship,  have  performed 
computational  research  that  indicates  that  this 
electromagnetic-acoustic  (pressure)  interaction  can  be 
exploited  to  help  identify  materials.  Banks  and 
colleagues  have  worked  at  microwave  frequencies  and 
have  shown  that  an  acoustic  wave  front  can  act  as  a 
weakly  reflective  moving  mirror  within  a  medium  such 
as  soil.  Using  short  microwave  pulses  and  exploiting  the 
vast  difference  between  electromagnetic  speeds  in 
materials  and  acoustic  speeds,  acoustic  wave  fronts  can 
be  interrogated  as  they  spread  in  a  structure. 
Interrogating  the  acoustic  or  pressure  fronts  appears  to  be 
able  to  aid  in  the  characterization  of  the  material  between 
the  surface  of  the  object  and  the  “collision  point”  of  the 
acoustic  wave  front  and  the  microwave  pulse. 

This  research  is  aiming  at  exciting  military 
applications.  Specifically,  it  is  expected  that 
underground  bunkers  will  have  air  conditioning  units  or 


motorized  airflow  systems.  These  constitute  a  localized 
acoustic  source  that  may  be  located  using  microwave 
scattering  off  of  the  outwardly  moving  acoustic  waves. 
The  research  will  also  investigate  whether  vehicles  with 
motors  running  have  acoustic  signatures  that  will  permit 
identification  in  settings  of  high  clutter.  Finally,  the 
earth  has  naturally  occurring  vibratory  events.  This 
research  will  ask  whether  scattering  of  these  naturally 
occurring  events  can  aid  in  the  imaging  of  ground  and 
sub-surface  targets. 

2.2  Mammalian  Vision  as  Processing  Model 

The  need  for  fast  accurate  scalable  approaches  to  image 
fusion  has  partially  motivated  AFOSR  investments  in 
modeling  of  the  human  visual  system  and  other 
mammalian  systems  of  similar  complexity.  For  example, 
These  efforts  are  motivated  by  knowing  that  human 
image  processing  for  recognition  of  objects  in  complex 
scenes  is  efficient,  taking  just  a  few  hundred  milliseconds 
in  a  few  stages  of  neural  processing.  Processing, 
however,  is  characterized  by  complex  local  non-linear 
neural  connections  with  plenty  of  feedback.  Some  of  this 
complexity  is  evident  in  Figure  2,  a  drawing  of 
mammalian  retina  provided  to  illustrate  neural 
connections  for  initial  multi-band  registration  of  images, 
in  three  types  of  cone  cell  on  the  left,  and  the  following 
adaptive  contrast  enhancement  and  encoding  into 
separate  chromatic  image  streams. 


Figure  2:  Retina  provides  multi-band  encoding  for 
fusion 

processing 

AFOSR  focuses  on  discovering  algorithms  based  on 
models  of  these  and  later  stages  and  their  algorithmic 
support  for  human  and  computational  processing  of 
complex  scenes.  Models  are  expressed  formally  and 
developed  into  complex  neural  network  architectures 
with  important  features  for  image  processing.  Such 
encoding  and  processing  appears  very  efficient,  as 
described  for  the  natural  system,  and  is  robust  with 
respect  to  desired  task  performance.  For  example,  the 
retinal  encoding  represented  in  Figure  2  supports  a 
variety  of  human  visual  tasks:  motion  processing  for 
navigation  and  computation  of  depth  maps;  stereo 


processing  for  position  and  depth  maps  in  support  of 
reaching;  texture  processing  for  material 
characterization;  contour  detection  and  completion  for 
edge  typing  and  surface  identification;  and,  of  course,  the 
segmentation  of  objects  from  their  backgrounds  and  their 
rapid  recognition  and  identification. 

Retinal  processing  proceeds  at  multiple  spatial  scales 
and  temporal  resolutions  through  organized  sets  of 
spatio-temporal  and  chromatic  filters  defined  by  their  so- 
called  weighting  functions.  In  general,  these  image 
weighting  functions  and  their  interactions  are  defined 
locally  and.  as  a  result,  bear  some  similarity  to  current 
wavelet  based  approaches.  (According  to  reports,  the 
first  patent  on  wavelet-like  self-similar  basis  sets  for 
image  processing  was  awarded  almost  twenty  years  ago 
to  a  researcher  in  human  vision).  Image  fusion 
techniques  based  on  discoveries  in  retinal  and  cortical 
models  of  vision  tend  to  proceed  from  coarse  to  fine 
spatial  scales.  Additional  models  based  on  cortical 
processing  add  many  important  features  to  the  retinal 
models,  including  algorithms  for  precise  registration  of 
contours  that,  depending  on  the  spectral  band  of  an 
image,  may  be  offset  or  missing. 

AFOSR  supported  models  of  mammalian  visual 
processing  have  proven  useful  in  the  contexts  described 
above.  Algorithms  for  stereo  vision  have  been  employed 
to  create  depth  maps  used  to  guide  reaching  and  grasping 
in  robotics  applications.  Algorithms  that  process  image 
sequences  have  proven  useful  for  collision  avoidance  in 
autonomous  navigation.  Algorithms  for  texture  detection 
and  discrimination  have  proven  useful  in  contexts  of 
material  classification  and,  with  continued  work,  may 
prove  useful  in  terrain  characterization.  Algorithms  for 
contour  detection  and  completion  are  beginning  to  merge 
with  algorithms  for  shape  encoding  for  identification. 
Algorithms  for  learning  and  classification  of  image  and 
other  data  have  proven  useful  in  a  variety  of  contexts  as 
will  be  described  later.  Lastly,  models  of  image  encoding 
have  inspired  novel  techniques  for  false  color  display  of 
multi-spectral  images.  Approaches  to  these  techniques  of 
image  display  are  outlined  next. 

2.3  False  Color  Display  of  Fused  Imagery 

Data  representations  for  fused  images  are  important  to 
the  Air  Force  in  a  number  of  applications.  Those  for 
automatic  recognition  are  discussed  in  later  sections. 
Flere,  displays  for  human  operators  are  emphasized.  The 
human  operator,  whether  on-line  wearing  a  head- 
mounted  display  device  or  off-line  performing  tasks  of 
image  analysis,  represents  a  large  and  growing  consumer 
of  fused  imagery.  For  example,  the  U.S.  Congress  has 
called  the  need  for  additional  scheduled  launches  of 
reconnaissance  satellites  into  question  because  the 
number  of  image  analysts  available  seems  insufficient  to 
process  the  increased  volume  of  image  data.  The 
AFOSR  program  in  Fluman  Performance  is  considering 
ways  to  address  the  need  for  increased  capacity  in  image 
analysis.  The  approach  under  discussion  would  supply 
the  analyst  with  a  more  refined  image  stream,  one  pre- 


processed  according  to  rules  discovered  by  study  of  the 
experts  involved. 

The  most  straightforward  approach  to  false  color 
display  of  multi-spectral  imagery  involves,  roughly 
speaking,  driving  each  of  the  three  phosphors  in  a  color 
display  device  with  an  image  stream  obtained  in  a 
different  spectral  band.  An  approach  similar  to  this  is 
taken  in  an  AFOSR-supported  effort  to  assess  ways  that 
operators  benefit  from  novel  head-mounted  devices  that 
display  polarimetric  images  superimposed  on 
monochromatic  images  of  the  same  scene.  As  described 
above,  such  fused  displays  may  be  useful  in  breaking 
camouflage  of  manufactured  items  such  as  tanks  or  other 
ground  vehicles. 


Register  Color 

via  3D  Model  Display 

Figure  3:  Image  fusion  architecture  based  on  neural 
models  of  human  vision 


Models  of  mammalian  visual  processing  have  been 
adapted  for  contrast-enhanced  display  of  multi-band 
image  streams  derived  from  visible,  infrared,  radar  and 
other  image  types,  such  as  polarimetric.  Alan  Waxman, 
for  example,  has  derived  a  number  of  architectures  for 
multi-band  image  processing  and  display.  Typically, 
each  includes  three  stages  of  post-processing  as 
illustrated  in  Figure  3.  The  first  stage  involves 
normalization  and  contrast  enhancement  within  a  band, 
processes  similar  to  those  of  the  retina.  The  second 
stage,  which  can  be  hierarchical  as  shown,  involves 
processing  between  bands,  in  a  fashion  similar  to  that  of 
“double  opponent”  processing  in  human  color  vision. 
This  second  stage  helps  to  locally  de-correlate  the  image 
streams  and  can  be  used  to  create  new  streams  from 
weighted  versions  of  multiple  input  streams.  Lastly,  the 
resulting  images  are  mapped  to  drive  intensities  in  a 
color  display. 

The  architecture  shown  in  Figure  3  is  one  example  of 
several  developed  by  Alan  Waxman,  the  results  of  which 
will  be  demonstrated  in  a  later  paper  [3].  Architectures 
have  been  developed  for  more  than  three  input  streams, 
and  some  have  been  implemented  in  hardware  for  feed¬ 
forward  processing  at  speeds  exceeding  the  frame 


interval.  In  summary,  such  techniques  offer  a  fast 
scalable  approach  to  image  fusion 
No  standard  multi-band  processing  or  display  color¬ 
mapping  scheme  is  yet  available.  The  human  factors  of 
such  displays  are  just  beginning  to  receive  attention. 
Converging  on  a  set  of  standards  for  image  fusion  and 
display  may  benefit  the  Air  Force  through  savings  in 
bandwidth  and  speed  of  processing.  For  example,  the 
fused  products  of  multi-band  images  might  be 
transmitted  from  sensor  platforms  instead  of  the  raw 
image  streams.  For  example,  lossy  compression  schemes 
might  be  developed  based  on  the  needs  of  human  viewers 
reflected  in  the  models  of  human  visual  processing.  An 
extreme  example  of  such  compression,  also  studied  under 
AFOSR  support,  involves  trading  field  of  view  for  spatial 
resolution  in  displays  with  pixel  resolution  that  declines 
from  the  image  center  to  match  the  decline  in  spatial 
resolution  of  the  human  retina.  Such  displays  may  have 
use  for  human  in  the  loop  applications  where  size  and 
weight  of  processing  and  display  hardware  is  at  a 
premium. 


2.4  Category  Recognition  and  Learning 

AFOSR  has  supported  theory  and  modeling  of  human 
pattern  recognition  and  decision  making,  informed  by 
human  and  animal  experiments  and  descriptions  of 
information  processing  in  cortex  and  other  brain  regions. 
For  example,  a  family  of  neural  network  models  of 
category  learning  and  recognition  developed  by  Gail 
Carpenter  at  Boston  University  has  proven  useful  in  a 
variety  of  applied  contexts,  including  pattern  recognition 
in  multi-spectral  fused  imagery.  One  example  [4]  from 
the  family  of  architectures  based  on  Adaptive  Resonance 
Theory  is  illustrated  below. 

The  pattern  recognition,  feed-forward,  aspect  of  the 
neural  architecture  illustrated  in  Figure  4  can  be 
essentially  described  as  a  pattern  classifier,  to  define 
clusters  in  the  space  of  input  features,  topped  off  by  a 
system  for  mapping  these  recognized  patterns  to 
equivalence  classes  under  the  application  of  interest. 
During  feedback  learning,  the  weighted  inputs  to 
definitional  nodes  for  patterns  and  for  classes  are 
adjusted  according  to  rules  derived  from  theories  of 
cognition  and  learning.  Learning  at  the  pattern 
recognition  level  can  proceed  automatically,  under 
management  of  a  parameter  that  determines  precision  of 
match  in  the  feature  space.  Learning  in  the  classifier 
proceeds  under  supervision,  say,  of  a  human  expert  in  the 
application  domain. 


Recognized  Class  Training  Class 


Target  Search  ^ 

Learn  Target 

oc 

T_J 

?9°-) 

N  Class 

Z  match? 

OC 

y__: 

«5 

-0 

0 

-\  Pattern 

J  match? 

o 

o 

^ _ 

*  1  A  *  i 

booooo 

* 

Input  Feature  Patterns 
Multi-Sensor  Processed  Imagery 


Figure  4:  ARTMAP  architecture  for  recognition  learning 

Neural  architectures,  such  as  ARTMAP,  based  on 
theories  of  Stephen  Grossberg  [5],  have  proven  capable 
of  fast  stable  learning  from  large  noisy  data  sets  where 
the  classes  of  interest  are  arbitrarily  defined.  Application 
domains  include  engineering  design  reuse,  3D  object 
recognition,  sensory  motor  control  and  navigation,  and 
others,  including  satellite  remote  sensing.  For  example, 
a  current  project  on  satellite  remote  sensing,  not 
supported  by  AFOSR,  involves  use  of  Landsat  data  and 
terrain  data  to  identify  classes  of  vegetation,  their 
mixtures  and  their  changes  over  time.  ARTMAP 
performance  in  this  domain  exceeds  the  speed  of  human 
experts  and  appears  to  match  them  in  performance. 

Alan  Waxman  and  colleagues  have  begun  to  use  the 
ARTMAP  architecture  to  learn  and  subsequently 
recognize  classes  of  objects  defined  in  the  feature  space 
of  multi-spectral  images.  For  example,  the  fused  images 
generated  according  to  processes  described  in  the 
previous  section  are  supplied  to  analysts  who  identify 
image  regions  containing  examples  (and 
counterexamples)  of  objects  of  interest.  Subsequent  to 
learning,  ARTMAP  processing  of  new  images  can  be 
used  to  highlight  image  regions  with  similar  features. 
The  use  of  ARTMAP  as  an  aid  to  image  analysis  will  be 
illustrated  in  the  later  talk  of  Dr.  Waxman  [6]. 

2.5  Shape  Coding  and  Matching 

AFOSR  also  invests  in  the  study  of  ways  that  humans 
learn  and  recognize  image  shapes.  The  goal  of  this 
research  is  to  provide  image-processing  systems  with  the 
accuracy  and  the  flexibility  of  human  observers  while 
greatly  increasing  the  speed  of  processing  compared  to 
the  human  benchmark.  The  topic  of  shape  coding  is 
germane  to  this  meeting  because  desired  object  coding 
schemes  must  be  robust  with  respect  to  multi-spectral 
data  in  which  the  generation  of  shadows,  occlusions,  and 
missing  contours  may  follow  different  physical  laws.  Of 
equal  importance  is  the  potential  benefit  of  using 


information,  such  as  contour  information,  from  one  band 
to  constrain  computations  on  images  from  another  band. 

AFOSR  does  also  invest  in  research  topics  that  may 
precede  human  image  coding  and  support  it.  These 
topics  are  omitted  from  discussion  in  the  interest  of 
brevity.  Such  topics  include  contour  identification  and 
completion,  surface  characterization  and  completion, 
shape  or  shape  parts  from  structure  or  motion  or  texture 
or  parallax  or  shading  etc.,  and  depth  from  stereo.  These 
topics  are  important,  and  may  need  to  be  resolved  in  the 
context  of  multi-spectral  imaging,  but  are  more  central  to 
discussion  at  meetings  more  directly  concerned  with 
automatic  target  detection  and  recognition. 

The  demonstrations  of  Dr.  Waxman,  however,  point 
the  way  to  object  detection  and  recognition  in  fusion 
contexts.  In  those  demonstrations,  image  regions 
containing  multi-spectral  features  associated  with  targets 
can  be  identified  and  associated  with  target  classes  of 
interest.  This  approach,  then,  may  provide  effective 
cueing  for  shape  recognition  algorithms  that  can  operate 
on  the  fused  image  data. 

AFOSR  supports  research  on  shape  recognition  and 
shape  matching.  One  example  is  provided  in  the  work  of 
Steven  Zucker  at  Yale  University.  This  approach  uses 
bounded  image  contour  regions  as  input,  and  initiates  a 
diffusion  process  that  produces  shocks  along  curves 
(Blum’s  medial  axis)  interior  to  contour-defined  regions. 
The  shocks  are  classified  into  four  types  determined  by 
the  shape  of  surrounding  contours,  as  shown  in  Figure  5. 
The  shock-defined  curves  of  an  object  connect,  and  the 
pattern  of  connections  can  be  expressed  as  a  graph  to 
provide  a  complete  skeletal  description  of  the  object. 
Experiments  of  Zucker  and  colleagues  [7]  have 
demonstrated  that  the  skeletal  graphs  provide  a  fast  and 
effective  means  of  generating  shape  descriptions  from 
images  and  comparing  them  to  stored  descriptions  for 
recognition.  These  experiments  also  demonstrate  that 
the  shape  comparisons  based  on  skeleton  graphs  appear 
robust  with  respect  to  scaling,  rotation,  deformation  (e.g. 
perspective),  and  occlusion. 

This  approach  may  lead  to  image  processing 
technologies  of  interest  to  the  Air  Force  in  applications 
where  fast  and  accurate  identification  of  objects  is  a 
premium.  For  example,  in  surveillance  or  targeting  of 
large  numbers  of  ground  objects  -  which  may  be  similar 
(but  differ  in  value,  e.g.  tanks/trucks,  friend/foe),  or 
articulated  (e.g.  having  slewable  or  extendable  parts),  or 
partially  occluded  or  camouflaged  -  both  image  analysts 
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and  missile  seekers  may  benefit  from  object-based 
templates  few  in  number  that  support  a  similarity  metric. 
Figure  5:  Four  shock  types,  primitives  for  skeletal  object 
descriptions 

The  approach  to  object  template  matching  described 
above  is  based  on  object  2D  outlines  or,  equivalently, 
object  silhouettes.  Interestingly,  2D  silhouettes  may 
partially  account  for  object  recognition  in  human 
observers.  Recent  work  of  Patrick  Cavanaugh  and 
colleagues  at  Flarvard  University  has  demonstrated  that 
object  templates  based  on  silhouettes  (more  precisely,  2- 
tone  images  that  may  contain  object  information  in  light 
or  dark  regions)  may  be  sufficient  to  account  for  human 
recognition  of  familiar  objects.  In  a  series  of  experiments 
[8],  this  group  demonstrated  that  recovery  of  object  parts 
from  image  data  is  not  required  for  object  recognition  in 
2-tone  images.  Recognition  of  familiar  object  is  fast  and 
veridical.  An  example  is  provided  in  Figure  6. 


Figure  6:  Familiar  and  unfamiliar  arrangements  of  same 
volumetric  parts 

Additional  AFOSR-supported  research  is  underway  to 
determine  how  an  object's  2D  contours  might  be  reliably 
acquired  from  noisy  image  data  with  occlusions  and 
shadows  that  may  interfere  with  algorithmic  attempts  to 
produce  object  outlines.  Fused  sensor  imagery  may  ease 
this  task.  In  combination,  however,  the  work  described 
above  provides  some  optimism  that  2D  templates, 
relatively  few  in  number,  may  provide  a  robust  method 
for  fast  and  accurate  identification  and  graded 
discrimination  of  highly  similar  objects. 

2.6  Adaptive  Control 

The  AFOSR  supports  research  on  dynamical  control  of 
distributed  parameter  structural  systems  for  space  and 
flight  control  systems  for  air  vehicles.  Among  other 
approaches,  these  programs  seek  understanding  of 
processing,  computation,  and  control  found  in  biological 
systems  simple  enough  to  enable  system-level 
descriptions  of  them,  yet  complex  enough  to  display 
performance  characteristics  not  fully  understood. 


Figure  7 :  Sensory-motor  fusion  for  adaptive  control 

Such  research  is  expected  to  contribute  to  development 
of  new  technologies  for  autonomous  operation  of 
uninhabited  air  vehicles,  munitions,  and  satellites, 
working  a  variety  of  missions  perhaps  in  cooperating 
swarms. 

Past  AFOSR  supported  work  has  led  to  demonstration 
of  neural  architectures  that  merge  sensory  maps  of  the 
environment  with  motor  maps  of  the  local  space  to 
generate  essentially  a  look-up  table  for  motor  commands 
to  execute  movement  to  a  desired  location  from  any 
starting  position.  Further,  the  resulting  sensory  motor 
maps  support  a  degree  of  adaptation  to  changing  physical 
plant  due  to  loss  of  calibration,  damage,  changes  in  mass, 
or  other  perturbations. 

3  Summary 

The  AFOSR  supports  basic  research  supporting  all 
aspects  of  an  idealized  closed-loop  system  capable  of 
searching  complex  environments  for  objects  of  interest, 
deciding  appropriate  actions,  and  taking  effective  actions 
to  alter  the  environmental  landscape.  Each  of  these 
idealized  capabilities  relies  heavily  on  fusion  of 
information  and  data.  In  sensor  programs,  AFOSR 
attempts  to  discover  new  ways  to  acquire  multi-band 
images  in  a  single  device.  In  signal  processing  and 
cognitive  programs,  AFOSR  attempts  to  discover  new 
algorithms  for  processing  multi-sensor  data  in  support  of 
decisions  on  target  identification.  Lastly,  in  guidance 
and  control  programs,  AFOSR  attempts  to  discover  novel 
approaches  for  autonomous  control  of  air  and  space 
platforms. 

These  basic  science  programs,  of  course,  support  a 
number  of  approaches  but  in  each  is  found  some  study  of 
biological  systems.  Biological  systems  are  efficient,  so 
their  study  can  provide  very  good  information  about  what 
to  compute.  How  to  compute  can  take  full  advantage  of 
modern  technologies,  so  the  speed  and  flexibility  of 
biological  systems  might  be  greatly  exceeded  in 
implementation. 

The  scientific  programs  of  AFOSR  support  research  in 
both  foreign  and  domestic  institutions.  A  description  of 
AFOSR  programs,  and  links  to  Internet  sites  of  the 
foreign  liaison  offices  can  be  found  at 
http://www.afosr.af.mil. 
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